Skip to content

Simplify prediction API type hinting #61

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Mar 21, 2025

Conversation

ncoghlan
Copy link
Collaborator

@ncoghlan ncoghlan commented Mar 21, 2025

With the addition of GBNF grammar structured response format support, structured responses are no longer required to contain valid JSON.

Accordingly, type hinting for prediction APIs has been simplified:

  • PredictionResult is no longer a generic type
  • instead, parsed is defined as a str/dict union field
  • whether it is a dict or not is now solely determined at runtime (based on whether a schema was sent to the server, and the content field has been successfully parsed as a JSON response)
  • other types that were only generic in the kind of prediction they returned are also now no longer generic
  • prediction APIs no longer define overloads that attempt to infer whether the result will be structured or not (these were already inaccurate, as they only considered the response_format parameter, ignoring the structured field in the prediction config)

Closes #59

With the addition of GBNF grammar structured response format support,
structured responses are no longer required to contain valid JSON.

Accordingly, type hinting for prediction APIs has been simplified:

* `PredictionResult` is no longer a generic type
* instead, `parsed` is defined as a `str`/`dict` union field
* whether it is a dict or not is now solely determined at runtime
  (based on whether a schema was sent to the server, and the content
  field has been successfully parsed as a JSON response)
* other types that were only generic in the kind of prediction they
  returned are also now no longer generic
* prediction APIs no longer define overloads that attempt to infer
  whether the result will be structured or not (these were already
  inaccurate, as they only considered the `response_format` parameter,
  ignoring the `structured` field in the prediction config)
@github-actions github-actions bot added the CLA signed Indicates that all contributors have signed label Mar 21, 2025
@ncoghlan ncoghlan merged commit c81eea0 into main Mar 21, 2025
10 checks passed
@ncoghlan ncoghlan deleted the simplify-prediction-result-type-hints branch March 21, 2025 15:09
@github-actions github-actions bot locked and limited conversation to collaborators Mar 21, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
CLA signed Indicates that all contributors have signed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Proposal: simplifying structured prediction result type hinting
1 participant